DOCUMENT RESUME 



ED 431 488 



JC 990 417 



TITLE 

INSTITUTION 
PUB DATE 
NOTE 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Virginia Community College System. Summary Assessment 
Reports , 1995 . 

Virginia State Dept, of Community Colleges, Richmond. 
1995-00-00 

12p. 

Reports - Research (143) 

MFOl/PCOl Plus Postage. 

♦College Outcomes Assessment; *Community Colleges; 
♦Educational Assessment ; Educational Practices ; ♦Educational 
Quality; Institutional Evaluation; Program Evaluation; 

School Effectiveness; Student School Relationship; ♦Transfer 
Students; Two Year Colleges 
♦Virginia Community College System 



ABSTRACT 



Thi s report reviews the evaluation process of the 1995 
community college assessment reports for the Virginia Community College 
System (VCCS) . It provides evaluation results, which found that the 18 
colleges, even within limited resources and major restructuring, were able to 
demonstrate satisfactory progress as shown by assessment results and the uses 
they had made of them. Results are related by assessment category, including 
general education, majors , off-campus , other than dual -credit , but including 
distance learning, and dual credit instruction. A brief summary is presented 
of the college results, the improvements that colleges have made, and 
identification of what appears to be trends in assessing each category. Also 
discussed is the link between restructuring and student outcomes assessment. 

A list of examples is provided, demonstrating how assessment results have 
been used by colleges to meet VCCS ' s restructuring goals: (1) to improve 

student transfer, (2) to improve skills and the performance of students, (3) 
to improve effectiveness and reduce the costs of developmental studies, (4) 
to examine the purpose and structure of the general studies associate degree, 
(5) to revise technology to meet workplace needs, (6) to assist faculty in 
incorporating instructional technology approaches, (7) increasing enrollments 
in dual -credit, and (8) to increase the number of students who complete the 
associate degree. Discusses planning, evaluation, resource allocation, and 
college assessment processes. (VWC) 



******************************************************************************** 

♦ Reproductions supplied by EDRS are the best that can be made ♦ 

♦ from the original document . * 

******************************************************************************** 



JC 990 417 ED 431 488 



VIRGINIA COMMUNITY COLLEGE SYSTEM 

SUMMARY 

1995 ASSESSMENT REPORTS 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



B. A. Blois 



M DEPARTMENT OF EDUCATION 

Ifice bt Edu^Si”^ and Improvement 

Uci^lONAL RESOURCES INFORMATION 
CENTER (ERIC) 

This document has been reproduced as 
received from the person or organization 
originating it. 

Minnr r.hannss have been made to 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 





2 






X 



Virginia Community College System 
Summary 

1995 Assessment Reports 



The Evaluation Process 

On November 14th and 15th, a team of six evaluators, the Vice Chancellor for 
Academic Services and Research, and the Director of Educational Planning met in 
Richmond to evaluate the 1995 community college assessment reports. Six weeks prior 
to these discussion sessions, each of the six evaluators had received either eleven or 
twelve reports to read and evaluate the 1994 letters to the presidents, a diskette with a 
blank copy of the evaluation checklist, and information on how the evaluation sessions in 
Richmond would be conducted. Each evaluator was responsible for preparing an 
evaluation checklist for the reports they were assigned to read; to lead the discussion for 
3-4 reports; and to prepare an evaluative summary, reflecting the team consensus for 
these reports. 

Teams of four evaluators met in forty-five minute time segments to discuss each 
report. Team members were rotated so that four teams, composed of different members, 
evaluated the twenty-three reports. 

Six assessment coordinators volunteered to evaluate the 1995 reports; another 
coordinator new to the system participated as an observer. All of the evaluators have had 
experience in leading successful assessment programs at their colleges. They were well 
prepared and the report discussions were very thorough, with equal consideration being 
given to all of the major reporting topics. As a result, the process yielded evaluations that 
were objective and thorough, providing comments that should be useful to each college. 
The evaluators' comments are contained in the letters to the presidents and the evaluation 
checklists for each college that are attached to this report. 



Evaluation Results 

Even within limited resources and major restructuring, eighteen colleges, much to 
their credit, were able to demonstrate satisfactory progress as shown by assessment 
results and the uses they had made of them. Many colleges showed evidence of mature, 
cyclical, and smoothly fimctioning assessment programs. Yet, programs were not 
stagnant, and colleges seemed to be always looking for improved performance indicators 
and continuous faculty involvement, especially in interpreting findings. Principles of 
good assessment practice were evident throughout the satisfactory reports. The colleges 
used a mix of direct and indirect measures, internal and external evaluators, and both 
quantitative and qualitative evaluation processes. The effect of the creation of a viable 
assessment culture at most colleges has been that the colleges have used results to 
improve, in numerous ways, programs, services, teaching, and student learning. Student 
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outcomes assessment seems to have been especially relevant during the past two years as 
colleges have reduced the length of programs, met Southern Association requirements for 
institutional effectiveness, restructured, and maintained a vigilance on quality assurance. 
During the past two years, good assessment practice by two colleges garnered 
commendations from the Southern Association for institutional effectiveness, which 
includes student outcomes assessment. A third college received a commendation in this 
area the previous year. 

The evaluations of three colleges showed that they had underachieved during this 
reporting period. Coincidentally, one of the three had a change in the instructional dean, 
another in the president, and the third had both a change in the president and instructional 
dean. Two colleges were deemed to be non-achievers and similarly, one of these had a 
change in president; at the remaining college, the institutional researcher/assessment 
coordinator resigned. 

To address the problems experienced by each of these colleges, the Chancellor or 
Vice Chancellor of Academic Services and Research will meet with the presidents, 
instructional deans or both, as appropriate, to discuss each leader's future plans for 
improving the assessment program. 



Results by Assessment Category 

This year colleges began reporting on an annual schedule, covering fewer 
assessment topics each year, but still fulfilling all SCHEV reporting obligations over the 
course of every two years. This year’s reports included assessment results in general 
education; majors; off-campus, other than dual-credit, but including distance learning; 
and dual-credit. A brief summary of the college results, the improvements that colleges 
have made, and identification of what appear to be trends in assessing each category are 
presented below. 

General Education. Good performance indicators for achievement in general 
education continues to be a challenge to most colleges. As a direct measure, 
colleges generally rely on commercial, nationally normed tests such as the 
Academic Profile. Others are the College Assessment of Academic Proficiency 
(CAAP) and the CAAP Essay Writing Test. One college, with the help of two 
psychology professors, has invested a great deal of time and developed its own 
general education test, and is in the process of ensuring its validity and reliability 
through pilot testing at other colleges. 

The nationally normed tests showed this year, and as they have over time, that 
Virginia's community college students perform at equivalent levels to freshmen 
and sophomores at two-year and four-year colleges nationwide. As indirect 
indicators of student achievement, colleges continue to use standard tests such as 
the Community College Student Experiences Questionnaire, and to survey 
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employers, current students, and graduates. Some have used focus groups and 
advisory committee members to determine whether students have the requisite 
skills to be successful in the workplace. One trend that the evaluators observed 
was that most colleges seem to be continuously revising objectives so that they 
are more meaningful to faculty, but also more measurable. Colleges are also 
vigilant that the general education objectives are taught in all areas of the 
curriculum. One college was commended by the evaluators for its course audit 
process that included a thorough review of the extent to which general education 
is infused throughout the curriculum. 

Colleges appear to be streamlining the assessment of general education, 
addressing problems, in particular, that have been called to their attention by 
external evaluators who may also be employers. Many colleges have begun 
focusing on assessing communication skills, both oral and written; computation; 
computer skills; and critical thinking. They have added writing, speaking, and 
mathematics to many programs; i.e., an "across the curriculum" orientation. At 
one college, for example, engineering faculty are working with English faculty to 
include more writing assignments in engineering courses. Another noticeable 
trend has been the use of external readers of essays using criteria-based reviews. 
Colleges continue to be creative with assessment methods, using, for example, 
video tapes of final speeches and presentations of research projects to advisory 
committee members. 

Improvements have resulted from curricular changes, including reorganizing 
course content and the sequence of courses; establishing pre-requisites; changing 
textbooks and resource materials; providing more opportunities for oral 
expression through reduced class size; including additional writing assignments in 
all courses; changing exams to include more critical thinking; increasing academic 
lab support; and increasing attention to pedagogy and the use of technology. 



Majors. Studies of program length begun in 1994, in particular, required that 
colleges re-examine program objectives and requisite skills. Colleges reported 
that objectives were also revised to include performance factors that employers 
identified as important; to correct weak areas of student performance as identified 
through assessment data from previous years; and to ensure that student 
achievement of the objectives could actually be measured. Some colleges still 
struggle with establishing objectives that meet all of these criteria. 

Colleges use a mix of both direct and indirect measures. They select among a 
menu of standardized achievement tests; professional licensure exams; capstone 
course exams; portfolio analysis; student surveys and interviews; exit interviews; 
graduate surveys; employer surveys; jury reviews; and skills checklists. Applying 
direct measures is easier in some majors because of the availability of 
standardized achievement tests. This is true, for example, in health technology 
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programs, electronics, office technology, automotive technology, and accounting. 
Colleges tend to use more indirect measures than direct, but the 1 995 evaluators 
stressed, in their feedback to colleges, the importance of having at least one direct 
measure for each major. 

The use of outside evaluators is prevalent in assessing the majors. Colleges use 
practicing professionals and members of curricular advisory committees to 
evaluate portfolios, research projects, and taped speeches. Many colleges cite the 
use of external evaluators as fostering partnerships. One college stated that the 
"assessment process provides an atmosphere of community partnerships among 
faculty, area businesses, and graduates". Another college expanded this concept, 
noting that the close contact between curricular advisory committee members, 
business and industry, local public schools (dual-credit assessment), and the four- 
year institutions enhances curricular design. At the same time, these partnerships 
enhance the preparation of some students to meet changing business and industry 
needs and other students to succeed at four-year institutions. 

College were fairly thorough in reporting analysis and use of follow-up surveys of 
graduates of AAS programs. The reviewers were disappointed to note that few 
colleges integrated findings from transfer studies into their program analyses. 

The findings of assessment in the majors have lead to improvements that include 
reorganization of course content, deletion of courses, and resequencing courses, as 
well as establishing prerequisites; changing textbooks and resource materials; 
increasing attention to pedagogy and the use of technology; purchasing updated 
equipment and software; and encouraging students to enroll in cooperative 
education and internship programs. 

Off-Campus, Including Distance Learning. In the 1993 report, many colleges 
reported that they were still in the planning stages of assessing off-campus courses 
and programs. Since that time, a majority of colleges have finished developing 
their plans and begun implementing them. Most colleges have focused on 
comparing on-campus and off-campus grade point averages, grade distributions, 
and individual course grades, where the same classes are offered both on and off 
campus, but taught by the same faculty. 

Colleges are aware that where adjunct faculty teach off-campus classes, there is 
potential for more variability in the achievement of established learning 
objectives. Where differences in grade distributions in the same courses are found 
between regular and adjunct faculty, division chairs have been prompt to discuss 
expectations with adjunct faculty. Colleges report that this process is generally 
successful and promotes course equivalency. One college has an exemplary 
process that helps to assure the equivalency of on-campus and off-campus 
courses. The college requires that each adjunct faculty create a portfolio that 
includes the course syllabus, all tests and handouts, and a sample of graded work 
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submitted by students. In turn, the portfolio is evaluated by the program head and 
the division chair, with results being discussed with the assessment coordinator. 
Most colleges also use off-campus student evaluations of courses and faculty, and 
their perspectives are also reflected in college-wide student surveys and graduate 
exit surveys. 

College reports indicate that, overall, quality and student achievement in off- 
campus courses are equivalent to those offered on-campus. In instances where 
differences have been found in student performance, the colleges appear to have 
been quick to address the related problems through consultation between division 
chairs and individual faculty. 

Distance Learning. The 1995 report is the first one in which colleges have 
reported on assessing distance learning. Assessment in this area is yet maturing 
and is complicated by the fact that colleges are using various delivery methods for 
this form of instruction which is neither time nor place bound. Colleges are in the 
process of planning for major changes in instructional delivery, while at the same 
time realizing the critical need, as with traditional off-campus instruction, to 
assure quality and instructional equivalency for courses. They are, nonetheless, 
faced with developing and implementing evaluative models for highly varied 
instructional methods and numerous settings. To cite some examples, instruction 
may be delivered through the use of compressed video, audio and video 
conferencing, or extended learning institutes. It may be delivered in on-campus or 
off-campus settings where students must be physically present, or students may 
study at home, and come to campus infrequently, or not at all. A third factor that 
complicates the evaluation of the comparability of distance learning with on- 
campus courses concerns whether faculty and students actually are in each other's 
presence during classes, or whether they communicate electronically, for example. 

The evaluators of the 1995 assessment reports observed that the reports on 
assessing distance learning showed that assessment in this area is in its early 
stages. Colleges are now using broad parameters to evaluate their distance 
learning programs. They monitor important but general data trends and obtain 
some measures of satisfaction among students and faculty, and other information 
from which to judge the adequacy of services. Colleges appear, within the context 
of assessing distance learning, to have gained a greater appreciation for assessing 
the outcomes of student services. Most colleges are carefully monitoring factors 
such as student demographics (to evaluate changes in access); enrollment trends; 
non-completion and graduation rates; grade distributions and grade point averages 
for distance learning and on-campus instruction; student satisfaction; faculty 
satisfaction; faculty qualifications; course materials; instructional support services 
such as library and learning resources; facilities; and the adequacy and proper 
functioning of technology. Student evaluations of courses are another indicator 
used by most colleges. Several colleges noted that these evaluations are 
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especially valuable because many of the students enrolling in off-campus or 
distance learning courses are also simultaneously taking classes on campus. 

Feedback in these areas has enabled colleges to make improvements in 
counseling, orientation, and first class meetings; voice mail and computer bulletin 
board systems; computing laboratories; course selection; and placement testing as 
well as course testing procedures. By monitoring student progress, colleges have 
a better idea of the types of students who will be successful in distance learning 
courses, and the kinds of changes in support services that need to be made to 
ensure student success. This is important groundwork for keeping withdrawal 
rates at a minimum, ensuring equivalency of courses, and for refining assessment 
processes in the future. 

In written comments provided by evaluators, colleges have been encouraged to 
use more direct indicators of student performance in assessing distance learning. 
Evaluators suggested that colleges consider some of the same methods that have • 
been useful in assessing off-campus and dual-credit instruction. These methods 
include cross-grading projects; capstone projects; standard test questions coupled 
with item analysis; the use of external evaluators; and on-site visits. Although 
more attention needs to be given to the use of direct indicators in assessing 
distance learning, the evaluators were convinced that the colleges were giving 
serious attention to ensuring the comparability of off-campus and distance 
learning instruction with the instruction on campus. 

Dual-Credit Instruction. This was the first year that colleges have been required 
to report fully on assessing dual-credit instruction. As with other off-campus 
instruction (approximately 82% of dual-credit instruction was offered at high 
schools in fall 1993), course equivalency is a primary issue for assessment 
purposes. The results provided by most colleges in the 1995 report dealt with the 
following questions: How have dual-credit students performed in courses? How 
does their performance compare with college students enrolled in the same 
courses? How have dual-credit students performed at community colleges and 
four-year institutions? Do students rate dual-credit courses as more or less 
challenging than other college-level courses they have taken? 

Many colleges observed that dual-credit students are achievers by nature. They 
were not, therefore, surprised when they found that the grades of these students 
were higher than the grades of students taking the same course on campus. In 
instances where the dual-credit grades were not equivalent or higher, most 
colleges have consulted with faculty, both regular and dual-credit, asking them to 
provide explanations and solutions to problems. One advantage of assessing dual- 
credit instruction as well as other off-campus instruction is that problems are 
identified each term and usually resolved fairly quickly. 
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Because so many courses are taught by high school faculty (approximately 83% 
in fall 1993), colleges seem to be investing a great deal of time, first, in ensuring 
that they have the appropriate credentials (18 graduate credits in the major field). 
Their next concern is ensuring that the dual-credit courses are, indeed, taught at 
the college level. This orientation derives not only from the large percentage of 
high school faculty teaching dual-credit courses, but also from some surveys 
indicating that students did not find dual-credit courses challenging. 

Using these findings, several colleges reported that they have developed faculty 
mentoring programs. One college has done exemplary work in this area, requiring 
that all adjunct faculty produce a portfolio of work for each class they teach. The 
portfolio is evaluated by full-time faculty teaching the same course. At this 
college, problems were discovered in two English composition courses. Full-time 
English faculty chose to address the problem by creating a "model" portfolio 
containing papers graded according to full-time faculty standards. The model 
portfolios were sent to all dual-credit faculty and will be used to ensure that grade 
inflation does not occur in dual-credit classes. 

Many colleges assured evaluators that they require equivalent syllabi and texts for 
dual-credit classes, and that division chairs visited dual-credit classes at least once 
each term. Overall, colleges were specific in stating that placement requirements 
for dual-credit students were the same as those required for other students. 

Several colleges, however, did not discuss their placement policies. This matter 
requires further investigation by the VCCS. 

Colleges had the most difficulty when they attempted to evaluate the performance 
of dual-credit students at four-year institutions. This problem is exacerbated by 
the fact that a high percentage of students take a dual-credit course at a 
community college, but go directly to a four-year institution after graduation. 
Most colleges indicated that the next important step in assessing dual-credit 
students is to obtain specific and current data on student performance at four-year 
institutions. State Council staff have agreed to work collaboratively with System 
Office staff to extract data in a usable form from the statewide database 
maintained by the Council. Such data are crucial if community colleges are to 
improve dual-credit student assessment. 



Restructuring and Student Outcomes Assessment 

A natural link between assessment and VCCS restructuring initiatives has 
occurred as colleges over the past two years have reduced the length of associate degree 
programs. Colleges were asked to study all of their associate degree programs, following 
a design that included a review of educational objectives for each program (the first step 
in good practice for assessing majors), and determining the minimal number of credits 
that could be offered to achieve the objectives. During this process, colleges relied 
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heavily upon various assessment data to eliminate duplicate courses, determine 
appropriate prerequisites, and re-sequence courses within credit ranges established by the 
VCCS. They used assessment results (such as those from employer surveys and licensing 
exams, and analysis of transfer data), input from advisory committee members who had 
been directly involved in evaluating student performance, and external accreditation 
standards as guides in reducing the length of programs. Such information and assessment 
data analyses enabled each college to re-design programs, focusing on the skills and 
competencies that will enable students to succeed on the job and at four-year institutions. 

Colleges stated that the effect of program streamlining will be determined, over 
time, as they analyze assessment data and seek explanations if, and when, changes in 
student performance occur. We anticipate that because faculty have been so immersed in 
program length reductions, they will be careful stewards and the best evaluators on this 
issue. Some colleges plan to ask faculty to identify the effects of streamlining in their 
program evaluation reports. 

Several colleges are anticipating that one effect of streamlining will be that the 
number of graduates will increase. Others observed that in the future it may be more 
difficult to correct weaknesses identified in student performance because simply adding a 
course is no longer an option. 

Included in the 1995 assessment reports were numerous other examples of how 
assessment results have been used by colleges to meet V CCS restructuring goals. A 
detailed listing is beyond the scope of this report, but some typical examples are provided 
below. 

* To improve student transfer, colleges have analyzed transfer data, finding in 
several instances, that more AAS degree students are transferring. Using 
results such as these, colleges have developed more articulation agreements, 
improved advising, and changing courses to fit four-year institution 
requirements. 

* To improve skills and the performance of students, colleges have used 
feedback from employers arid other assessment data to identify deficiencies, 
then revised courses to include more writing, speaking, mathematics, critical 
thinking, and greater use of computers and library resources. 

* To improve effectiveness and reduce the costs of developmental studies, 
colleges have tracked student performance over time, analyzed data on student 
and faculty perceptions of preparation for college-level courses, and made 
greater use of technology and tutors. 
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* To examine the purpose and structure of the general studies associate degree, 
colleges have identified courses that do not transfer, evaluated the success of 
graduates who transfer to four-year institutions, and looked at their 
standardized general education test scores. 

* To revise technology to meet workplace needs, colleges are relying more on 
advisory committee and external evaluator input, surveying graduates and 
their employers yearly, and monitoring student scores on standardized tests to , 
ensure mastery of technical skills important in changing work environments. 

* To assist faculty in incorporating instructional technology approaches into 
teaching, colleges will continue to give special attention to student evaluations 
of courses, retention rates, and grades, especially in distance learning classes 
and those introducing computer use in the classroom. Instructional 
technology training programs and technology support centers are being 
initiated, and stipends are provided through the professional development 
initiative to encourage faculty participation in training. Some colleges have 
obtained grants to support faculty development. 

* The restructuring goal of increasing enrollments in dual-credit has signaled 
the increase in college interest in improving assessment of dual-credit students 
previously described. Between fall 1993 and fall 1995, enrollment in dual- 
credit courses has increased from 3,710 to 5,329 students. 

* To increase the number of students who complete the associate degree, 
colleges use assessment data obtained at various points in a student's program, 
as in capstone courses, to monitor their progress and intervene at appropriate 
times. Other assessment data show, for example, that the longer students stay 
at the colleges, the greater their success at four-year institutions. Colleges also 
seek information from non-completers and many are now making changes 
based on assessment of student services functions. 



Planning, Evaluation, Resource Allocation and College Assessment Processes 

Many reports assured evaluators that the colleges had been successful in 
integrating planning, evaluation, and resource allocation processes. They describe 
strategic planning processes where assessment findings are routinely used for 
environmental scanning, thereby identifying issues that need to be included in college 
planning strategies. Some colleges describe faculty and staff retreats where assessment 
results are used to identify future goals, determine their priority from a college-wide 
perspective, and establish the human and financial resources required to achieve the 
goals. One assessment coordinator reported that she has been named as a member of 
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three of the college's four governance committees in order to identify issues related to 
student achievement and supply related outcomes data to inform committee decisions. 

At some colleges, assessnient results are integrated into planning, evaluation, and 
resource allocation primarily through the program review process. In these instances, 
monies are often provided to programs that demonstrate through assessment findings, for 
example, a need for updated equipment. Colleges appear to be making deliberate changes 
to ensure that assessment increasingly effects resource allocation decisions. One 
president, in an annual memorandum to budget managers, requests that they identify 
budget implications including staffing, training, equipment, facilities, learning resources, 
and supplies, based upon the results obtained from assessment. Another college has 
shifted its program evaluation schedule to correspond to its budgeting calendar. The 
report of this college also provides examples where assessment results have contributed 
to hiring additional faculty, increasing program budgets, upgrading laboratories, 
implementing an automated placement testing system, and increasing professional 
development activities. 

Although the community colleges continue to use assessment results for 
planning, evaluation, and resource allocation, the paradox is that almost none of the 
colleges have, in the face of restructuring and current fiscal constraints, had the luxury of 
retaining a full-time assessment coordinator. Many assessment coordinators have other 
duties that often are not closely related to assessment. One large college kept its full-time 
assessment coordinator, but the trade-off was a reduction in the assessment budget. The 
point here is that restructuring clearly has forced many colleges to minimize the resources 
devoted to assessment programs. Along with SCHEV, we have always maintained that 
the number of FTE assigned to assessment is less important than whether colleges can 
deliver the results. The conclusion we draw from our careful review of this year’s 
reports, is that almost all our colleges are “delivering the goods.” The colleges are to be 
commended for their ability to maintain assessment programs under continuing and 
severe fiscal restraints. 
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